Stylus based haptic peripheral for touch screen and tablet devices

ABSTRACT

A system, method, and computer program product for automatically providing a haptic stylus based interface for enhanced user interaction with touch screen devices are disclosed. The stylus communicates with the touch screen device to report stylus operating information from onboard sensors, and to receive application program instructions defining feedback and friction forces to be applied by onboard actuators. The applied feedback and friction forces may be independently controlled. Friction forces are provided via a rolling contact ball gripping mechanism that mimics varying physical engagement between the contact ball and a touch screen display surface by controlling the relative rotational freedom of the contact ball. Embodiments of the invention enable a haptic stylus to mimic three-dimensional interactions for gaming, object manipulation and sculpting, and non-contact proximity-based stylus operation scenarios.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to three other simultaneously-filedapplications, all with a common specification, including applicationSer. No. 13/112,860 entitled “Haptic Device for 3-D Gaming”, applicationSer. No. 13/112,909 entitled “Haptic Device for Carving and MoldingObjects”, and application Ser. No. 13/112,932 entitled “Haptic Devicefor Position Detection”, each of which is hereby incorporated byreference in its entirety. Commonly-assigned patent application U.S.Ser. No. 12/948,472 filed on Nov. 17, 2010, is also hereby incorporatedby reference in its entirety.

FIELD OF THE INVENTION

The present patent document relates in general to input devices forcomputing environments, more specifically to hand-held stylus-type toolsfor interfacing with touch screen devices.

BACKGROUND OF THE INVENTION

Touch screen devices are becoming more common, being used currently forexample in cellular telephones, digital cameras, personal digitalassistants (PDAs) and other handheld computing or gaming devices. Touchscreen user interfaces typically combine a display unit capable ofdepicting visual output with an overlying touch sense unit capable ofdetecting user input via touch. Touch screens lack some of thefunctionality of more conventional input devices, so many also include astylus for inputting data and/or navigating a user interface. Stylusdevices for use with portable touch screens have become very familiar toa broad user base.

Haptic devices employ tactile feedback technology that takes advantageof a user's sense of touch by conveying physical force sensations to theuser. The resulting vibration or motion can be employed to enhance theuser's perception of information being presented by a computer. Suchhaptic devices are used for example in video game controllers (e.g. theSony PS3 DualShock®), and complex 3-D editing tools for computer-aideddesign systems. The use of haptic peripherals with portable touch screencomputing devices has not become widespread to date, though.

As a result, there is a need for a haptic hand-held stylus interfacetool to improve the interaction between users and their portable touchscreen computing devices.

SUMMARY OF THE EMBODIMENTS

A system, method, and computer program product for haptic interactionwith a touch screen display are disclosed and claimed herein. Theembodiments generally assist a user by more convincingly simulating themanipulation of objects depicted on the display.

Embodiments of a haptic stylus interface device may comprise a styluscapable of providing feedback forces and having a contact ball capableof rotating in place, a communications module enabling informationtransfer with a computer, and a contact ball gripping mechanism thatresponsively produces a variable contact ball friction force independentof the feedback forces. The device may further comprise at least onesensor that measures operational data including at least one ofposition, orientation, velocity, acceleration, and rotation. The sensormay include at least one of a gyroscope, an accelerometer, and a contactpressure sensor.

The device may further comprise a feedback actuator that selectivelyproduces the feedback forces oriented with respect to a housing, eitherlongitudinally or orthogonally, or some combination of the two. Thefeedback actuator may press a contact pad against the contact ball fortransmission of feedback forces. The contact pad may be slick orfrictional in nature. The device may be inverted for use as a trackball.

The communications module, preferably a Bluetooth module, includescontrol circuitry that gathers the operational data for transmission,and receives control information enabling responsive actuatoractivation. Driver software executing on the computer manages theoperational data and the control information for an applicationexecuting on the computer. The device may transfer information by wireor wirelessly. The contact ball may roll against the operating surfaceof a touch screen display.

The contact ball gripping mechanism may comprise a set of prongs holdingthe contact ball in a grip of variable firmness, including completelystopping contact ball rotation. The various components of the device maybe placed in a housing or handle that may be cylindrical or conical inshape, with the contact ball and contact ball gripping mechanismpreferably at the larger end of the cone. The stylus may includeconventional user input devices as well, including a button, a knob, awheel, and a sliding switch for example.

Alternate embodiments may include a computer interface for providinginteraction between a computer and a user, comprising a computer and ahaptic stylus that communicates electronically with the computer andprovides independent feedback and friction surface contact forces to auser. The invention may also be embodied as a method for providinginteraction between a computer and a user, comprising transferringinformation between a computer and a stylus having a contact ballcapable of rotating in place, and selectively providing independentfeedback and friction forces to a user with the stylus. Finally, theinvention may be embodied as a computer program product comprising amachine-readable medium tangibly embodying non-transitory programinstructions thereon that, when executed by a computer, cause thecomputer to transfer information between a computer and a stylus havinga contact ball capable of rotating in place, and selectively provideindependent feedback and friction forces to a user with the stylus.

Further, embodiments may enable a gaming interface that helps a userinteract with a game program executing on the computer, by simulatinggame objects or characters. In this case, the effort applied by the useris determined from stylus pressure or stylus motion against the frictionforce, for example. Excess stylus pressure may damage the touch screendisplay, so a warning may be given should this eventuality be detected.Rotary motion of the stylus may trigger a predetermined game event.Sudden changes in the friction force may indicate particular gameevents; likewise a steady increase in the friction force may indicatethat a game character is fatigued or damaged.

Embodiments of the invention may enable sculpting operations on virtualor real objects by selectively adding or removing object material. Inthis case, the application program executing on the computer is asculpting or three-dimensional CAD application. The friction force mayprovide information to the user regarding the object's properties, e.g.different regions of the object may have their own frictioncharacteristics. For example, embedded knots or hard objects may bemodeled by a sudden increase in the friction force when they areencountered by a manipulation tool or sculpting implement. Similarly,softer or empty regions such as voids may be modeled by a suddendecrease in the sculpting implement friction force. Objects may havedirectional friction properties as well, used to model wood grain forexample, or to help beginning sculptors recognize regions to be shavedaway to produce a desired object. Overcoming various friction forcesand/or exerting stylus contact pressure are ways for the user to specifyan effort level when sculpting, which may control the amount of materialbeing manipulated. Finally, the stylus system may take advantage of aproximity detecting tool that determines the distance from the contactball to its contact surface, typically the operating surface of a touchscreen display.

As described more fully below, the apparatus and processes of theembodiments disclosed permit automatic haptic interaction with a touchscreen display. Further aspects, objects, desirable features, andadvantages of the apparatus and methods disclosed herein will be betterunderstood and apparent to one skilled in the relevant art in view ofthe detailed description and drawings that follow, in which variousembodiments are illustrated by way of example. It is to be expresslyunderstood, however, that the drawings are for the purpose ofillustration only and are not intended as a definition of the limits ofthe claimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a prior art stylus and handheld portable touch screenelectronic device;

FIG. 2 depicts a stylus according to an embodiment of the presentinvention;

FIG. 3 depicts a stylus according to another embodiment of the presentinvention; and

FIG. 4 depicts a flowchart of stylus operation according to embodimentsof the present invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Referring now to FIG. 1, a prior art stylus and handheld portableelectronic device with a touch screen display are shown. The stylus isnot a haptic device in this instance. The stylus provides a convenienthandle for grasping, as well as a precise point of contact with theportable electronic device's touch screen display. In essence, it ismerely a more precise version of a user's finger. Such a stylus istypically housed inside the portable electronic device when not in use.

Referring now to FIG. 2, stylus 200 is shown, according to an embodimentof the present invention. In this embodiment stylus 200 includes aconical housing 202, though other embodiments may employ alternatelyshaped housings, e.g. cylindrical. Housing 202 is designed to serve as afamiliar and effective handle for a user to grasp during manipulation ofthe stylus. This embodiment also includes a contact ball 206 capable ofrotating in place, and a contact ball gripping mechanism including forexample a group of prongs 204 moved by a friction force actuator (notshown). Any type of actuator may be employed; a wide variety ofelectromechanical actuators are known in the art for example. Note thatcontact ball 206 is shown protruding extensively from housing 202 forclarity in this patent application, but generally may be much closer tohousing 202 and/or partially contained within a spherical indentation ofhousing 202.

During operation, the user holds the stylus housing 202 by hand andpresses contact ball 206 against a surface, such as the smooth surfaceof a touch screen display. The contact point between contact ball 206and the touch screen display surface is determined in that case by thetouch screen display circuitry, as with a conventional stylus. The usermay easily move contact ball 206 around on the surface, as contact ball206 is normally not constrained in its rotation; its distance fromhousing 202 does not usually vary noticeably though.

However, the contact ball gripping mechanism may selectively apply acontrollable gripping force to contact ball 206 to provide hapticfeatures. The friction force actuator varies the mechanical grippinglinkage between for example housing 202 and contact ball 206, to controlthe contact ball's relative freedom of movement. In other words, prongs204 may selectively loosen or tighten their effective grip on thecontact ball. The prongs shown are hinged, but unhinged prongs may alsobe employed.

This braking feature controls the frictional drag between contact ball206 and the rest of stylus 200. The result is that additional force isrequired from the user to overcome the friction between prongs 204 andcontact ball 206. This friction is however generally intuitivelyinterpreted by the user as being between contact ball 206 and the touchscreen surface, even though such a surface may be quite slick.

The contact ball gripping mechanism may even completely stop contactball 206 rotation, so that the force required to drag contact ball 206across the display screen surface is maximized. Contact ball 206 is thuspreferably made of an elastomeric material for more effective grippingand dragging, though that is not a required limitation. As shown, fourprongs 204 may be employed, though any number may suffice; a singleretracting ring, a pair of retracting semi-circular or arc-shapedprongs, or three symmetrically arranged prongs 204 for example (notshown) are all within the scope of the invention as they retain contactball 206 and influence its freedom of rotation.

Stylus 200 may also be inverted to serve as a trackball, wherein theuser manipulates a possibly larger contact ball 206 directly, againstthe friction force that may be generated as described. Stylus 200 mayalso include conventional user input devices such as a button, a knob, awheel, and a sliding switch (not shown) for example. Other componentscommon between this embodiment and that of FIG. 3 are now described.

Referring now to FIG. 3, stylus 300 according to another embodiment isshown. This stylus features a cylindrical housing 302, a contact ball306, and a contact ball gripping mechanism including prongs 304 andfriction force actuator 316 that controls them as previously described.As with FIG. 2, this stylus is preferably powered by a battery 314.

Communications module 308, sensors 310, and feedback actuator 312 areshown in this figure but are also common to the embodiment of FIG. 2.The communications module is preferably a Bluetooth wireless modulepositioned at the top of housing 302 (though wired communication meansare also within the scope of the present invention) that enablesinformation transfer with a computer, such as the portable electronicdevice having a touch screen display. The communications module acquiresstylus operational data (to be described) gathered by onboard controlcircuitry and transmits it to the computer. The communications modulealso receives control information from the computer that may govern thetype and timing of haptic feedback to be provided by the stylus. Forexample, the computer may indicate when contact ball 306 is to be freeto rotate and when it is not, and to what extent its rotation is to beinhibited by friction actuator 316 and prongs 304.

The stylus operational data may describe all aspects of the stylus thatcan be reported out, such as its position, orientation, velocity,acceleration, rotation, and whether its contact ball is currentlytouching anything and if so how forcefully. The operational data ismeasured by sensors 310 within housing 302 as shown in FIG. 3 (alsoemployed but not shown in FIG. 2) and provided to communications module308 by control circuitry (not shown). The sensors may be of any number,and of any type known in the art, such as but not limited to a3-dimensional gyroscope, an accelerometer, and a contact pressuresensor.

In addition to the friction forces described above, the stylusembodiments of FIGS. 2 and 3 may also generate feedback forces for theuser. For example, in response to received control signals, feedbackactuator 312 may selectively move a mass within the stylus. Battery 314may be the mass that is moved for example longitudinally within housing302; motion in other directions is also within the scope of theinvention. In this manner, application programs may better interact withthe user by linking the manipulation of objects depicted on the touchscreen display with the haptic feedback provided by the stylus.

Feedback actuator 312 may exert a force on contact ball 306 to forexample pull or push stylus 300 toward or away from the touch screendisplay surface temporarily. The feedback actuator may for example pusha contact pad (not shown) from housing 302 against the contact ballduring creation of the feedback force. Such bumps are commonly used toprovide the user with a sensation of “height” when a stylus pointtraverses the edges of graphical user interface objects on the touchscreen display, for example. This action may tend to inhibit therotational freedom of contact ball 306 however, by pushing it intoprongs 304, particularly if the contact pad is not a very low friction(i.e. slick) surface. The feedback force and the friction force aretherefore interrelated.

To avoid this issue, the embodiments of the present embodiment enableselectively independent control of the feedback forces and the frictionforces exerted by the stylus. Friction forces may be preciselycontrolled by the contact ball gripping mechanism. Any contribution tofriction forces caused by the feedback forces operating to press thecontact ball against the prongs can be predicted and/or measured, andthe intended friction forces can be responsively lessened to counteractthose contributions. For example, by lessening the grip the contact ballgripping mechanism has on the contact ball during the time the feedbackactuator is exerting a downward force on the contact ball, the netintended friction force is maintained. Similarly, if the feedbackactuator is retracting a contact pad away from the contact ball, thecontact ball will have space to move slightly away from the prongs,resulting in a more free rotation than was intended. To counteract this,the prongs may be moved toward the contact ball to grip it more tightlyuntil the feedback force abates.

Referring now to FIG. 4, a flowchart describing the overall operationalflow of embodiments of the present invention is described. In step 410,the stylus sensors acquire operational data describing the operatingparameters of the stylus, such as position, orientation, velocity and soforth. Next, in step 420 the control circuitry provides the operationaldata to the communication module for transmission to the computer thatis in communication with the stylus. In step 430, driver softwareexecuting on the computer receives the stylus operational data andconverts it to relevant descriptive interface data that has meaning toan application program. For example, contact pressure and stylusacceleration information may be converted into a touch screen displaylocation being touched by the contact ball, and particular manner inwhich the contact ball is moving, perhaps indicative of user intent.Then, in step 440, an application program executing on the computerreceives the interface data and determines what if any haptic forcesshould be applied by the stylus as a consequence. For example, theapplication program may decide to trigger a feedback force “bump”sensation and tighten the grip on the contact ball as it “ascends” anedge of a displayed object visually portrayed as having height, toprovide the illusion that the contact point has collided with and is“climbing up onto” the object.

In step 450, the driver software translates these desired forces intocontrol data that will cause the various stylus actuators to generatethe feedback and friction forces designated by the application program.In step 460, the computer transmits this control data to thecommunications module. In step 470, the feedback and friction forceactuators responsively trigger to meet the requirements of theapplication program. In step 480, the user feels the applied forces andmay responsively alter the stylus configuration. This process may repeatso the embodiments of the invention provide an ongoing automatic hapticinteraction to improve the computer-human interface between theapplication program and the user. In another embodiment,computer-executable program instructions for implementing the methodabove are provided via a computer program product.

The embodiments of the invention thus improve the simulation of movementand manipulation of display objects as if in a three-dimensionalinteractive environment when actually only two-dimensional non-texturedtouch screen displays are being used. Embodiments may be employed in avariety of particular simulations, which are now described.

The stylus may be beneficial to users who are running game programs astheir applications. Games require input from a user typically to controlsimulated persons and objects, but users of touch screen display devicesmay not have the same input devices that are available on a regularpersonal computer or video game machine. Embodiments of the inventionmay therefore provide a level of input detail not previously availableon portable touch screen devices. For example, the stylus may provideoperational data denoting the pressure being applied by a user againstthe touch screen display to indicate an effort level intended, forexample in a combat or sports game. Excess pressure may also trigger awarning that the touch screen display may become damaged. Increasedfriction forces may denote that a game character is becoming fatigued ordamaged from a virtual battle. Similarly, rotary motions of the stylus,including stylus rotation and stylus revolution, even if not contactingthe display surface, may trigger a game event, such as a characteraction like turning an object around or looking in different virtualdirections. A sudden release of the friction force may be used toindicate that a game event has occurred, such as a door or container hasbeen pried open, or that a sword has sliced through an object.

The stylus may also be beneficial to users who are performing variousthree-dimensional manipulation tasks on rendered virtual objects. Forexample, one may “carve” a virtual object with the stylus. Theapplication program may be a CAD tool that depicts an object on thetouch screen display. The motion of the contact ball against thefriction force may denote the strength being applied by the user to acarving implement, which responsively controls the depth of materialremoved during each pass of the implement. Similarly, one may build up avirtual object by depositing material at a rate according to thefriction force opposed by the user.

Further, while carving an object, embedded knots or voids may beeffectively simulated by sudden changes in the friction force requiredto move the stylus contact ball through these anisotropic regions. Forexample, an embedded knot may present a sudden increase in the frictionforce, such that the user may wish to avoid carving through it. This canbe used for teaching sculpting, i.e. removal of unwanted material iseasy while removal of the desired embedded portion is more difficult.Similarly, a sudden release of the friction force may be used to denotethat a void has been found, or that a layer has been removed. Likewise,wood grain may be denoted by a directional dependence of the frictionforce.

Finally, the stylus may be used with stylus proximity detection tools tohelp mimic a three-dimensional surface or object even though the contactball is not contacting a real surface or object. See for examplecommonly-assigned patent application U.S. Ser. No. 12/948,472 filed onNov. 17, 2010. A stylus proximity detection tool, which may be in thestylus or in the touch screen display device for example, automaticallydetermines the distance from the contact ball to the display surface.That data is added to the operational data received from the stylus.Thus one may have position detection and much of the stylus interactiondescribed above on a virtual surface or volume without the contact balltouching the surface.

As used herein, the terms “a” or “an” shall mean one or more than one.The term “plurality” shall mean two or more than two. The term “another”is defined as a second or more. The terms “including” and/or “having”are open ended (e.g., comprising). Reference throughout this document to“one embodiment”, “certain embodiments”, “an embodiment” or similar termmeans that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least oneembodiment. Thus, the appearances of such phrases in various placesthroughout this specification are not necessarily all referring to thesame embodiment. Furthermore, the particular features, structures, orcharacteristics may be combined in any suitable manner on one or moreembodiments without limitation. The term “or” as used herein is to beinterpreted as inclusive or meaning any one or any combination.Therefore, “A, B or C” means “any of the following: A; B; C; A and B; Aand C; B and C; A, B and C”. An exception to this definition will occuronly when a combination of elements, functions, steps or acts are insome way inherently mutually exclusive.

In accordance with the practices of persons skilled in the art ofcomputer programming, embodiments are described below with reference tooperations that are performed by a computer system or a like electronicsystem. Such operations are sometimes referred to as beingcomputer-executed. It will be appreciated that operations that aresymbolically represented include the manipulation by a processor, suchas a central processing unit, of electrical signals representing databits and the maintenance of data bits at memory locations, such as insystem memory, as well as other processing of signals. The memorylocations where data bits are maintained are physical locations thathave particular electrical, magnetic, optical, or organic propertiescorresponding to the data bits.

When implemented in software, the elements of the embodiments areessentially the code segments to perform the necessary tasks. Thenon-transitory code segments may be stored in a processor readablemedium or computer readable medium, which may include any medium thatmay store or transfer information. Examples of such media include anelectronic circuit, a semiconductor memory device, a read-only memory(ROM), a flash memory or other non-volatile memory, a floppy diskette, aCD-ROM, an optical disk, a hard disk, a fiber optic medium, a radiofrequency (RF) link, etc. User input may include any combination of akeyboard, mouse, touch screen, voice command input, etc. User input maysimilarly be used to direct a browser application executing on a user'scomputing device to one or more network resources, such as web pages,from which computing resources may be accessed.

While the invention has been described in connection with specificexamples and various embodiments, it should be readily understood bythose skilled in the art that many modifications and adaptations of theinvention described herein are possible without departure from thespirit and scope of the invention as claimed hereinafter. Thus, it is tobe clearly understood that this application is made only by way ofexample and not as a limitation on the scope of the invention claimedbelow. The description is intended to cover any variations, uses oradaptation of the invention following, in general, the principles of theinvention, and including such departures from the present disclosure ascome within the known and customary practice within the art to which theinvention pertains.

What is claimed is:
 1. A haptic stylus interface device, comprising: astylus comprising a feedback force actuator configured to providefeedback forces, and having a surface contact ball capable of rotatingin place; a communications module enabling information transfer with acomputer; and a contact ball gripping mechanism that produces a variablecontact ball friction force independent of the feedback force actuatoraccording to control data received from the computer.
 2. The device ofclaim 1 further comprising a sensor that measures operational dataincluding at least one of position, orientation, velocity, acceleration,and rotation.
 3. The device of claim 1 further comprising a sensor thatincludes at least one of a gyroscope, an accelerometer, and a contactpressure sensor.
 4. The device of claim 1 further comprising a feedbackactuator that selectively produces the feedback forces oriented withrespect to a housing in at least one of a longitudinal direction and anorthogonal direction.
 5. The device of claim 1 wherein thecommunications module further includes control circuitry that gathersthe operational data for transmission, and receives control informationenabling responsive actuator activation.
 6. The device of claim 5further comprising driver software executing on the computer, thatmanages at least one of operational data and control information for anapplication executing on the computer.
 7. The device of claim 1 whereinthe information transfer is one of wired and wireless.
 8. The device ofclaim 1 wherein the communications module is a Bluetooth module.
 9. Thedevice of claim 1 wherein the surface is the operating surface of atouch screen display.
 10. The device of claim 1 wherein the contact ballgripping mechanism comprises a set of prongs holding the contact ball ina grip of variable firmness.
 11. The device of claim 1 wherein thecontact ball gripping mechanism selectively completely stops contactball rotation.
 12. The device of claim 1 wherein the device isconfigured for operation as a trackball.
 13. The device of claim 1wherein the device simulates the manipulation of objects depicted on adisplay.
 14. The device of claim 4 further comprising a housingcontaining a sensor, the feedback actuator, the communications module,and the contact ball gripping mechanism.
 15. The device of claim 14wherein the housing is one of cylindrically shaped and conically shapedwith a larger end having the contact ball and contact ball grippingmechanism.
 16. The device of claim 1 further comprising: a feedbackactuator that produces the feedback forces; and a braking pad that thefeedback actuator presses against the contact ball when friction forcesand longitudinal feedback forces are interrelated.
 17. The device ofclaim 1 further comprising user input devices including at least one ofa button, a wheel, a knob, and a sliding switch.
 18. A computerinterface for providing interaction between a computer and a user,comprising: a computer; and a haptic stylus that communicates with thecomputer and provides independent feedback and friction surface contactforces to a user according to control data received from the computer,enabling a game interface.
 19. A method for providing interactionbetween a computer and a user, comprising: transferring informationbetween a computer and a stylus having a contact ball capable ofrotating in place; and selectively providing independent feedback andfriction forces to a user with the stylus according to control datareceived from the computer.
 20. A computer program product comprising amachine-readable medium tangibly embodying non-transitory programinstructions thereon that, when executed by a computer, cause thecomputer to: transfer information between a computer and a stylus havinga contact ball capable of rotating in place; and selectively provideindependent feedback and friction forces to a user with the stylusaccording to control data received from the computer.